Skip to content

chore(dkg): remove summary's transcripts for remote subnets (1/4) - #11220

Merged
pierugo-dfinity merged 5 commits into
masterfrom
pierugo/summary/remove-transcripts-for-remote-subnets-1
Aug 25, 2026
Merged

chore(dkg): remove summary's transcripts for remote subnets (1/4)#11220
pierugo-dfinity merged 5 commits into
masterfrom
pierugo/summary/remove-transcripts-for-remote-subnets-1

Conversation

@pierugo-dfinity

@pierugo-dfinity pierugo-dfinity commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Since the work performed to speedup the latency of remote DKG requests, their results are delivered as part of data blocks instead of summaries. The transcripts_for_remote_subnets field in summaries is thus now unused. This PR begins the migration to remove it entirely. Once it is, the batch delivery can be rewritten more cleanly as we would know that summary heights never contain any consensus responses.

We cannot simply replace the field with a BackwardsCompatible<Vec<_>, _> (i.e. replace the Vec<_> with an Option<Vec<_>>) because the field is a repeated, and an empty vector is not differentiable from an absent one. In contrast, Some(vec![]) and None have different hashes. For that reason, we need to add a new sentinel field to the protobuf definition that will be set to true to explicitely indicate that the field is absent. The entire rollout can be reviewed by looking at the stacked PRs and is described below (together with justifying why upgrades & downgrades are safe).

  • V0 (current) <-> V1 (after this PR):
    • The goal of V1 is to keep the exact same representation (including hash) of V0's summaries while still understanding the sentinel and compute the same hash as V2 for V2's summaries.
    • Upgrade V0 -> V1: A V0's summary doesn't set the sentinel so the field deserializes as Some(vec![]) and the hash of an empty vector is appropriately computed.
    • Rollback V1 -> V0: a V1's summary doesn't set the sentinel just yet. V0 does not read it anyways and always includes an empty vector in the hash, same as V1.
  • V1 <-> V2 (#11221):
    • The goal of V2 is to actually stop hashing an empty vector, and instead just ignore it.
    • Upgrade V1 -> V2: V2 still understands Some(vec![]) so will compute V1's summary's hash identically. For its own summaries, it starts to fill the sentinel (because the transcripts are None).
    • Rollback V2 -> V1: V1 understands the sentinel and thus ignores the transcripts while computing the hash.
  • V2 <-> V3 (#11223):
    • The goal of V3 is to remove the field entirely.
    • Upgrade V2 -> V3: V3 doesn't know about the transcripts anymore (the field is removed). V2 created a summary with None transcripts meaning their hash was not included, good.
    • Rollback V3 -> V2: V3 unconditionally sets the sentinel to true, meaning V2 will interpret the transcripts as None and not try to hash them.
  • V3 <-> V4 (#11224):
    • The goal of V4 is to remove the introduced sentinel field.
    • Upgrade V3 -> V4: V4 will ignore the sentinel field set by V3.
    • Rollback V4 -> V3: V3 will deserialize a summary with an unset sentinel, but it does not read it anyways.

@github-actions github-actions Bot added the chore label Aug 19, 2026
@pierugo-dfinity
pierugo-dfinity force-pushed the pierugo/summary/remove-transcripts-for-remote-subnets-1 branch from eb46cbe to e334f07 Compare August 20, 2026 07:32
@pierugo-dfinity
pierugo-dfinity requested a lite review from Copilot August 20, 2026 09:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR is the first step of a staged migration to remove transcripts_for_remote_subnets from DkgSummary without changing summary hashing across replica versions, by introducing a protobuf presence marker and switching the Rust field to a backwards-compatible representation.

Changes:

  • Wrap DkgSummary::transcripts_for_remote_subnets in BackwardsCompatible<…> and update protobuf (de)serialization to preserve hash behavior across versions using a new sentinel marker.
  • Add try_from_proto_with to BackwardsCompatible to support conversions that aren’t expressed via TryFrom.
  • Adjust consensus handling/tests to treat the summary’s remote transcripts as optional (and effectively absent going forward).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
rs/types/types/src/consensus/dkg.rs Switch summary remote transcripts to BackwardsCompatible and add marker-aware protobuf conversions.
rs/types/types/src/backwards_compatibility.rs Document field-removal lifecycle and add try_from_proto_with helper.
rs/protobuf/src/gen/types/types.v1.rs Regenerate protobuf bindings to include the new optional marker field.
rs/protobuf/def/types/v1/dkg.proto Add transcripts_for_remote_subnets_removed sentinel (tag 16) and bump next id.
rs/consensus/src/consensus/batch_delivery.rs Only generate remote-DKG responses from summaries when transcripts are present.
rs/consensus/dkg/src/test_utils.rs Stop reading remote transcripts from summary blocks (return empty).
rs/consensus/dkg/src/lib.rs Update test expectation for the BackwardsCompatible summary field.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread rs/types/types/src/consensus/dkg.rs
@pierugo-dfinity
pierugo-dfinity marked this pull request as ready for review August 25, 2026 06:52
@pierugo-dfinity
pierugo-dfinity requested a review from a team as a code owner August 25, 2026 06:52
@zeropath-ai

zeropath-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

No security or compliance issues detected. Reviewed everything up to 060bd4b.

Security Overview
Detected Code Changes
Change Type Relevant files
Enhancement ► rs/consensus/dkg/src/lib.rs
    Update tests to expect empty transcripts_for_remote_subnets for local configs
► rs/consensus/dkg/src/test_utils.rs
    Adjust extraction of transcripts_for_remote_subnets from Summary for compatibility
► rs/consensus/src/consensus/batch_delivery.rs
    Guard remote transcript generation behind presence check of transcripts_for_remote_subnets
► rs/protobuf/def/types/v1/dkg.proto
    Bump next id and add transcripts_for_remote_subnets_removed marker to Summary
► rs/protobuf/src/gen/types/types.v1.rs
    Reflect new transcripts_for_remote_subnets_removed field in Summary
► rs/types/types/src/consensus/dkg.rs
    Change transcripts_for_remote_subnets type to BackwardsCompatible<Vec, true> and initialize accordingly
► rs/types/types/src/consensus/dkg.rs
    Adapt DkgSummary to use BackwardsCompatible for transcripts_for_remote_subnets and propagate marker
► rs/types/types/src/consensus/dkg.rs
    Update From<&DkgSummary> for pb::Summary to handle new marker and optional field
► rs/protobuf/src/gen/types/types.v1.rs
    Add transcripts_for_remote_subnets_removed marker field in generated Summary
► rs/types/types/src/consensus/dkg.rs
    Modify conversion to protobuf to emit empty vector when absent and set marker accordingly

Comment thread rs/consensus/src/consensus/batch_delivery.rs
Comment thread rs/protobuf/def/types/v1/dkg.proto Outdated
@pierugo-dfinity
pierugo-dfinity added this pull request to the merge queue Aug 25, 2026
Merged via the queue into master with commit 6887d7d Aug 25, 2026
40 checks passed
@pierugo-dfinity
pierugo-dfinity deleted the pierugo/summary/remove-transcripts-for-remote-subnets-1 branch August 25, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants